Learning in humans is a complex process composed of various aspects. One aspect is memorization of data such as faces, facts, letters, sounds, formulae, etc. Another aspect is ability comprehend, to recognize and to generalize notions from various sources such as books, lectures, television, Internet, etc. Finally, humans learn from accumulated knowledge and, by reasoning, arrive at conclusions about new, e.g., unseen information. In general, humans also use intuition, experience, creativity and other subjective criteria to perform intelligent tasks. Machine learning aims at automating intelligent tasks in computer systems using reasoning, to infer new information from old. Common goal is to replicate humans' pattern recognition and classification skills in computer systems.
A typical task addressed in the field of machine learning is classification. This is the process of assigning an input object to one class from a set of classes, for example, to determine whether a given email is “spam” or “non-spam”. Usually, classification of objects in computer systems is based on training data as input data. Training data contains samples or instances of objects assigned the correct classification. For example, a number of exemplary mails classified as “spam”. Information about the objects is also entered into the computer system. For example, objects may be formally represented by features, which constitute properties or characteristics of the objects. The set of features used to describe the objects is referred to as the feature space. Classified sample objects and their features are analyzed to learn decision criteria based on which the sample objects are put into certain classes. Example features that may be used to classify mails may be “sender”, “content” such as terms occurring in the mail, etc.
The complexity of the feature space imposes limitations on the performance of automatic classification. For example, selecting from the feature space those features that are important and have discriminative potential cannot readily be done. Reducing the feature space while also choosing the important and informative set of features for the classification can be a challenging and computationally expensive task.